Skip to content

test(spark): characterize PartitionPathUtils partition discovery and materialization#8783

Merged
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:spark-partition-path-utils-tests
Jul 16, 2026
Merged

test(spark): characterize PartitionPathUtils partition discovery and materialization#8783
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:spark-partition-path-utils-tests

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Rationale for this change

PartitionPathUtils (java/vortex-spark/src/main/java/dev/vortex/spark/read/PartitionPathUtils.java)
discovers Hive-style partition columns from file paths and materializes them as constant
column vectors on the read path. It currently has no test coverage, despite encoding
several subtle behaviors: URL decoding of keys and values, the
__HIVE_DEFAULT_PARTITION__ null sentinel, first-= splitting, type inference
precedence (int → long → double → boolean → string), and per-type parsing into
ConstantColumnVector. This continues the characterization-test series from #8770
(read-path ArrowUtils) and #8782 (write-path SparkToArrowSchema).

What changes are included in this PR?

A new PartitionPathUtilsTest covering all three public methods:

  • parsePartitionValues: extracts ordered key=value segments from paths; ignores
    segments without the key=value shape (including empty keys or values); URL-decodes
    both keys and values (New%20YorkNew York); splits on the first = so values
    may themselves contain =.
  • inferPartitionColumnType: infers Integer for int-width values, Long for
    wider integrals, Double for decimal-point values, Boolean for case-insensitive
    true/false, and falls back to String (including for null and the
    __HIVE_DEFAULT_PARTITION__ sentinel).
  • createConstantVector: null and the Hive default-partition sentinel produce null
    vectors; string/int/long/short/byte/boolean/float/double values are parsed into their
    target types; DateType parses as days-since-epoch (int) and both timestamp types as
    micros-since-epoch (long); unrecognized target types fall back to UTF8 strings.

This is a test-only change; no production code is modified. 19 tests, all passing
locally (:vortex-spark_2.12:test --tests 'dev.vortex.spark.read.PartitionPathUtilsTest').

What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

…materialization

Signed-off-by: jackylee <qcsd2011@gmail.com>
@jackylee-ch jackylee-ch force-pushed the spark-partition-path-utils-tests branch from 8a4cb75 to 5b652ef Compare July 16, 2026 09:16
@robert3005 robert3005 added the changelog/chore A trivial change label Jul 16, 2026
@robert3005 robert3005 merged commit d54f70d into vortex-data:develop Jul 16, 2026
79 of 80 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants